--- title: Bert Visualize keywords: fastai sidebar: home_sidebar summary: "Visualize masked language modeling transformer model" description: "Visualize masked language modeling transformer model" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %} {% raw %} 
from transformers import AutoModelForMaskedLM,AutoTokenizer
{% endraw %} {% raw %}
{% endraw %} {% raw %} 
model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased",use_fast=True)

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForMaskedLM were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['cls.predictions.decoder.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
{% endraw %}

A piece of sample text

{% raw %} 
text = """I must not [MASK].
Fear is the mind-killer.
Fear is the little [MASK] that brings total obliteration.
I will face my fear.
I will permit it to pass over me and through me.
And when it has gone past I will turn the inner [MASK] to see its path.
Where the fear has gone there will be nothing.
Only I will remain."""
{% endraw %} {% raw %}
{% endraw %} {% raw %}

class MLMVisualizer[source]

MLMVisualizer(model, tokenizer)

{% endraw %} {% raw %} 
vis = MLMVisualizer.from_pretrained("bert-base-uncased")

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForMaskedLM were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['cls.predictions.decoder.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
{% endraw %} {% raw %}
{% endraw %} {% raw %}

li[source]

li(x)

{% endraw %} {% raw %}

infer_logits[source]

infer_logits(vis, y_pred, mask)

{% endraw %} {% raw %}

predict_text[source]

predict_text(vis, text)

{% endraw %} {% raw %}

visualize[source]

visualize(vis, text)

{% endraw %} {% raw %}

visualize_result[source]

visualize_result(vis, result:Config)

{% endraw %} {% raw %} 
%%time
result = predict_text(vis,text)

CPU times: user 606 ms, sys: 25.1 ms, total: 631 ms
Wall time: 164 ms
{% endraw %} {% raw %} 
%%time
vis.visualize(text)

Masked language modeling visualization

CPU times: user 1.83 s, sys: 69.2 ms, total: 1.9 s
Wall time: 975 ms
{% endraw %}

Different size of model

{% raw %} 
model = AutoModelForMaskedLM.from_pretrained("google/electra-small-generator")
tokenizer = AutoTokenizer.from_pretrained("google/electra-small-generator",use_fast=True)

vis = MLMVisualizer(model,tokenizer)
{% endraw %} {% raw %} 
vis.visualize(text)

Masked language modeling visualization

{% endraw %}